Papers with task success rate
Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition (2020.acl-main)
Copied to clipboard
| Challenge: | Many studies have applied reinforcement learning to train a dialog policy . but modeling a real-world user simulator is challenging and requires domain expertise . |
| Approach: | They propose to build dialog policies with two agents as dialog agents to avoid building a user simulator beforehand. |
| Outcome: | The proposed method can build a system policy and a user policy simultaneously . it can achieve high task success rate through conversational interaction . |
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (D19-1)
Copied to clipboard
| Challenge: | HANNA is an interactive photo-realistic simulator that can help agents with navigation tasks . human assistants are rich external knowledge sources but may not be available all the time to provide guidance . |
| Approach: | They develop a photo-realistic mobile agent simulator that asks for help from humans . they use natural language and visual instructions to direct agents towards the goals . |
| Outcome: | The proposed approach can accomplish tasks more effectively than competing models . it can also predict its own chances of making future progress . |
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Agentic systems are becoming more capable of defining strategies, taking actions, and solving complex, multi-step tasks. |
| Approach: | They propose an automatic, dynamic, and easy-to-use evaluation framework that provides textual insights into agent behavior on three levels of granularity: system, trace, and node. |
| Outcome: | The proposed framework produces high-quality, data-driven, insightful feedback on system, trace, and node. |
Sentiment Adaptive End-to-End Dialog Systems (P18-1)
Copied to clipboard
| Challenge: | Existing methods to train dialog systems only consider semantic inputs and under-utilize other user information. |
| Approach: | They propose to include user sentiment in the end-to-end learning framework to make dialog systems more user-adaptive and effective. |
| Outcome: | The proposed system improves on a bus information search task with sentiment information. |
Guided Dialogue Policy Learning without Adversarial Learning in the Loop (2020.findings-emnlp)
Copied to clipboard
Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Julia Kiseleva, Maarten de Rijke, Shahin Shayandeh, Jianfeng Gao
| Challenge: | Reinforcement learning methods suffer from sparse and unstable reward signals . alternating training of dialogue agent and reward model can get stuck in local optima . |
| Approach: | They propose to decompose adversarial training into two steps to improve dialogue policy learning. |
| Outcome: | The proposed method achieves remarkable task success rate using both on-policy and off-poly reinforcement learning methods. |
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing GUI agents focus on enhancing the accuracy of individual actions and lack effective mechanisms for detecting and recovering from errors. |
| Approach: | They propose a framework that incorporates a backtracking mechanism to enhance the task completion capabilities of GUI agents by verifier, judger, and reflector components. |
| Outcome: | The proposed framework improves task success rate and step accuracy on Mobile3M and Auto-UI benchmarks. |
Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to automating complex tasks focus on reactive policies and focus on visual observations. |
| Approach: | They propose a framework that prioritizes natural language understanding and structured reasoning to enhance the agent’s global understanding of the environment by developing a task-oriented, refinable world model at the outset of the task. |
| Outcome: | The proposed framework outperforms existing approaches in simulated environments and on real mobile devices. |
Decoupling Strategy and Generation in Negotiation Dialogues (D18-1)
Copied to clipboard
| Challenge: | Recent work on negotiation trains neural models, but their end-to-end nature makes it hard to control their strategy. |
| Approach: | They propose a modular approach that decouples strategy and generation by coarse dialogue acts . they test their approach on a recently proposed DEALORNODEAL game . |
| Outcome: | The proposed approach can decouple strategy and generation without degeneracy. |
GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent (2025.acl-long)
Copied to clipboard
| Challenge: | GUI automation is a key challenge in dynamic environments. |
| Approach: | They propose a training-free GUI agent that integrates two mechanisms to explore trajectories in GUIs. |
| Outcome: | The proposed GUI-explorer shows significant improvements over existing agents. |
OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing efforts to build GUI agents focused on the autonomous mode have failed to address the problem of over-execution. |
| Approach: | They propose an adaptive GUI agent that predicts confidence levels at each interaction step and elicits adaptive interaction. |
| Outcome: | The proposed GUI agent outperforms existing models on a complex dataset and on established benchmarks. |
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021.findings-acl)
Copied to clipboard
| Challenge: | despite recent progress, learning new tasks through language instructions remains a challenging problem. |
| Approach: | They propose a hierarchical task learning approach that decomposes task learning into three sub-problems and a model that addresses each sub-probability in a unified manner. |
| Outcome: | The proposed model achieves the state-of-the-art performance on the AL-FRED benchmark . it decomposes task learning into three sub-problems and addresses them in a unified manner . |
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, Lihong Li
| Challenge: | Existing work on reinforcement learning has focused on single-turn tasks such as solving math problems. |
| Approach: | They propose a framework that learns directly from online interactions by asynchronously generating diverse trajectories, guided by binary rewards depending on task success. |
| Outcome: | Experiments on the WebArena-Lite benchmark show that the framework outperforms state-of-the-art methods and strong proprietary models. |
Attacking Vision-Language Computer Agents via Pop-ups (2025.acl-long)
Copied to clipboard
| Challenge: | Existing tools for analyzing and testing VLMs are lacking in understanding what types of attacks are possible and what types are effective. |
| Approach: | They propose to integrate pop-ups into existing agent testing environments to attack VLM agents by ignoring them. |
| Outcome: | The proposed attack success rate is 86% and decreases by 47% when integrating pop-ups into existing agent testing environments. |
ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratch (2025.emnlp-main)
Copied to clipboard
Jiawei Chen, Xinyan Guan, Qianhao Yuan, Mo Guozhao, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han
| Challenge: | Existing instruction data synthesis methods focus on single-turn instructions and neglect cross-turn coherence, resulting in context drift and reduced task completion rates. |
| Approach: | They propose a framework that constrains multi-turn instruction synthesis by explicitly modeling human conversational intent. |
| Outcome: | The proposed framework outperforms existing models trained on single-turn and multi-turn instruction datasets. |
Hierarchical Prompting Assists Large Language Model on Web Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle on processing complicated observations in interactive decision making tasks. |
| Approach: | They propose a hierarchical prompting approach that constructs an action-aware observation and a Summarizer prompt. |
| Outcome: | The proposed method outperforms the current state-of-the-art prompting mechanism by 6.2% on task success rate. |
Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for embodied reasoning are coarse-grained and expensive . branch-and-browse framework enables fine-grounded, memory-guided, and efficient multi-branch reasoning. |
| Approach: | They propose a framework that unifies structured reasoning-acting, contextual memory, and efficient execution. |
| Outcome: | The proposed framework achieves task success rate of 35.8% and reduces execution time by up to 40.4% relative to state-of-the-art methods. |
Learning API Functionality from In-Context Demonstrations for Tool-based Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Documentation is often missing, outdated, privatized, or inconsistent in tool-based agents. |
| Approach: | They propose a task that allows agents to learn API functionality directly from in-context demonstrations. |
| Outcome: | The proposed model improves the task success rate of an API-based agent by providing explicit function calls and natural language critiques. |
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations (2025.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art multimodal web agents can perform many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). |
| Approach: | They propose to build multimodal web agents for few-shot adaptability using human demonstrations to improve their generalization and adaptability. |
| Outcome: | The proposed framework enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations. |
MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Evaluating text generation capabilities of large language models (LLMs) is challenging, especially for low-resource languages where methods for direct assessment are scarce. |
| Approach: | They propose a framework that transforms existing benchmarks into conversational tasks and measures LLMs’ accuracies on those tasks. |
| Outcome: | The proposed framework correlates strongly with established benchmarks while enabling standardized comparisons across languages and models. |